Exact and Perturbation Solutions for the EnsembleDynamicsTodd

This paper presents two approaches to characterize the dynamics of weight space probability density starting from a master equation. In the rst, we provide a class of algorithms for which an exact evaluation of the integrals in the master equation is possible. This enables the time evolution of the density to be calculated at each time step without approximation. In the second, we expand earlier work on the small noise expansion to a complete perturbation framework. As an example application, we give a perturbative solution for the equilibrium density for the LMS algorithm. Finally, we use the perturbation framework to review annealed learning with schedules of the form (t) = 0=t p .